System, method and computer product for incremental improvement of algorithm performance during algorithm development

ABSTRACT

System, method and computer product for incremental improvement of algorithm performance during algorithm development. In one aspect of this disclosure, there is a system, method and computer readable medium that stores instructions for instructing a computer system, to assess the performance of an algorithm during development. In this embodiment, a design of experiments component establishes an acceptable number of experiments for analyzing incremental improvements of the algorithm. An experiment performance component runs the established number of experiments for the algorithm. A simulation component simulates the behavior of the algorithm using results from the experiment performance component.

FEDERAL RESEARCH STATEMENT

[0001] The US Government may have certain rights in this disclosurepursuant to Government Contract Number MDA 972-98-3-0002 awarded by theDefense Advanced Research Projects Agency (DARPA).

BACKGROUND OF INVENTION

[0002] This disclosure relates generally to algorithm development andmore particularly to assessing the performance of an algorithm atdifferent phases of development and assessing the impact that inputvariations have on the algorithm.

[0003] Generally, an algorithm undergoes several validations during thecourse of its development. Validation is the systematic evaluation ofthe algorithm performance. The evaluation includes performing reviewsand tests to ensure that functional requirements are complete andtestable. Generally, a review is a form of inspection that includesreviewing documentation to make sure that it supports operation,maintenance and future enhancements. In addition, the review includesdetermining whether established requirements, design concepts andspecification have been met. Generally, a test includes operating thealgorithm with real or simulated inputs to demonstrate that it satisfiesthe functional requirements. Typically, during a test, various testcases having selected input data and parameters are used as inputs tothe algorithm. Outputs and reactions of the algorithms are then noted.Test failures are used to identify the specific differences betweenexpected and actual results.

[0004] There are several drawbacks associated with the above-notedprocess for validating algorithms. Most notably, when the choice ofalgorithms to be implemented is not known a priori, for example becausethe algorithms are tasked with solving a new problem that has no knownbenchmark solution, the impact of newly developed parts of the algorithmon solving the stated problem cannot easily be judged. In addition, ifthe problem domain is poorly known, algorithmic solutions are carriedwithout the benefit of knowing exactly the impact on the unknown partsof the problem domain. This predicament is compounded by several factorssuch as increasing complexity of the problem domain, multiple boundaryconditions, and permutation of possible cases.

[0005] In order to overcome the above drawbacks, there is a need for amethodology that allows a software developer to assess the performanceof an algorithm at different phases of development and assess the impactthat input variations will have on the algorithm.

SUMMARY OF INVENTION

[0006] In one aspect of this disclosure, there is a system, method andcomputer readable medium that stores instructions for instructing acomputer system, to assess the performance of an algorithm duringdevelopment. In this embodiment, a design of experiments componentestablishes an acceptable number of experiments for analyzing thealgorithm. An experiment performance component runs the establishednumber of experiments for the algorithm. A simulation componentsimulates the behavior of the algorithm using results from theexperiment performance component.

[0007] In another aspect of this disclosure, there is a system, methodand computer readable medium that stores instructions for instructing acomputer system, to assess the performance of an algorithm duringdevelopment. In this embodiment, a design of experiments componentestablishes an acceptable number of experiments for analyzing thealgorithm. An experiment performance component runs the establishednumber of experiments for the algorithm. A simulation componentsimulates the behavior of the algorithm using results from theexperiment performance component. A simulation performance componentevaluates the performance of the simulation for the algorithm.

[0008] In a third aspect of this disclosure, there is a system, methodand computer readable medium that stores instructions for instructing acomputer system, to assess the performance of an algorithm duringdevelopment. In this embodiment, a design of experiments componentestablishes an acceptable number of experiments for analyzing thealgorithm. An experiment performance component runs the establishednumber of experiments for the algorithm and uses a performance metric toevaluate the results of the experiments. A Monte Carlo simulationcomponent simulates the behavior of the algorithm using results from theexperiment performance component with a Monte Carlo simulation. Asimulation performance component evaluates the performance of the MonteCarlo simulation for the algorithm.

[0009] In still another aspect of this disclosure, there is a system,method and computer readable medium that stores instructions forinstructing a computer system, to assess the performance of an algorithmduring development. In this embodiment, a design of experimentscomponent establishes an acceptable number of experiments for analyzingthe algorithm. An experiment performance component runs the establishednumber of experiments for the algorithm. A performance metric componentevaluates the results of the experiments run for the algorithm. Analgorithm adjustment component adjusts logic or parameters of thealgorithm for unacceptable results.

[0010] In a fifth aspect of this disclosure, there is a system, methodand computer readable medium that stores instructions for instructing acomputer system, to assess the performance of an algorithm duringdevelopment. In this embodiment, a Monte Carlo simulation componentsimulates the behavior of the algorithm with a Monte Carlo simulation,wherein the Monte Carlo simulation component uses at least one confusionmatrix to simulate the behavior of the algorithm. A simulationperformance component evaluates the performance of the Monte Carlosimulation for the algorithm. An algorithm adjustment component adjustslogic or parameters of the algorithm for unacceptable results.

BRIEF DESCRIPTION OF DRAWINGS

[0011]FIG. 1 shows a schematic diagram of a general-purpose computersystem in which a system for assessing the performance of an algorithmoperates;

[0012]FIG. 2 shows a top-level component architecture diagram of analgorithm performance assessment system that operates on the computersystem shown in FIG. 1;

[0013]FIG. 3 shows an architectural diagram of a system that implementsthe algorithm performance assessment system shown in FIG. 2;

[0014]FIG. 4 shows a flow chart describing actions performed by thealgorithm performance assessment system shown in FIG. 2 as it operateswithin the system shown in FIG. 3;

[0015]FIG. 5 shows an example of a possible application for thealgorithm performance assessment system shown in FIG. 2;

[0016]FIG. 6 shows a fusion algorithm process map for the informationfusion tool shown in FIG. 5; and

[0017]FIG. 7 illustrates an example of simulating a diagnostic tooloutput for the information fusion tool shown in FIG. 5 with thealgorithm performance assessment system shown in FIG. 2.

DETAILED DESCRIPTION

[0018] This disclosure describes a system, method and computer productfor assessing the performance of an algorithm during its development.FIG. 1 shows a schematic diagram of a general-purpose computer system 10in which a system for assessing the performance of an algorithmoperates. The computer system 10 generally comprises a processor 12,memory 14, input/output devices, and data pathways (e.g., buses) 16connecting the processor, memory and input/output devices. The processor12 accepts instructions and data from memory 14 and performs variouscalculations. The processor 12 includes an arithmetic logic unit (ALU)that performs arithmetic and logical operations and a control unit thatextracts instructions from memory 14 and decodes and executes them,calling on the ALU when necessary. The memory 14 generally includes arandom-access memory (RAM) and a read-only memory (ROM), however, theremay be other types of memory such as programmable read-only memory(PROM), erasable programmable read-only memory (EPROM) and electricallyerasable programmable read-only memory (EEPROM). Also, memory 14preferably contains an operating system, which executes on the processor12. The operating system performs basic tasks that include recognizinginput, sending output to output devices, keeping track of files anddirectories and controlling various peripheral devices.

[0019] The input/output devices may comprise a keyboard 18 and a mouse20 that enter data and instructions into the computer system 10. Also, adisplay 22 may be used to allow a user to see what the computer hasaccomplished. Other output devices may include a printer, plotter,synthesizer and speakers. A communication device 24 such as a telephoneor cable modem or a network card such as an Ethernet adapter, local areanetwork (LAN) adapter, integrated services digital network (ISDN)adapter, Digital Subscriber Line (DSL) adapter or wireless access card,enables the computer system 10 to access other computers and resourceson a network such as a LAN, wireless LAN or wide area network (WAN). Amass storage device 26 may be used to allow the computer system 10 topermanently retain large amounts of data. The mass storage device mayinclude all types of disk drives such as floppy disks, hard disks andoptical disks, as well as tape drives that can read and write data ontoa tape that could include digital audio tapes (DAT), digital lineartapes (DLT), or other magnetically coded media. The above-describedcomputer system 10 can take the form of a hand-held digital computer,personal digital assistant computer, notebook computer, personalcomputer, workstation, mini-computer, mainframe computer orsupercomputer.

[0020]FIG. 2 shows a top-level component architecture diagram of analgorithm performance assessment system 28 that operates on the computersystem 10 shown in FIG. 1. The algorithm performance assessment systemcomprises a Design of Experiments (DoE) component 30 that establishes anacceptable number of experiments for analyzing an algorithm underdevelopment. The DoE is a well-known technique for mathematicallyselecting and analyzing experimental results to efficiently gatherinformation relating a process response to selected independentvariables. Generally, the DoE comprises steps such as defining aproblem, establishing an objective, selecting response variable(s),selecting independent variable(s), choosing variable levels, selectingan experiment design, running the experiment and collecting data,analyzing the data and drawing conclusions. As one of ordinary skill inthe art will recognize, these steps are dependent upon the particularapplication of the algorithm that is being developed.

[0021] In this disclosure, the DoE component 30 establishes theacceptable number of experiments based upon a selected subset of adesign space defined by an expert. In this disclosure, the design spaceis the space described by the variables and parameters or logic that areinput to the algorithm to be designed. The logic or parameters can bethe output from systems that are part of the process before thealgorithm to be designed. The acceptable number of experimentsdetermined by the DoE component 30 generally depends on the level ofaccuracy desired and is further constrained by resource limitations suchas financial resources or time resources. For example, using the DoEcomponent 30 allows tailoring the number of solutions that the experthas to evaluate to the expert time available with a trade-off onaccuracy. The DoE component 30 also selects an appropriate level ofexperiments for analyzing the algorithm. Generally, there are threelevels of DoE; 1) full factorial, 2) fractional factorial or 3)screening. A full factorial experiment allows all combinations of allvariables to be tested, while a fractional (e.g., half ) factorialexperiment allows many factors to be assessed simultaneously withreasonable economy of effort and a screening DoE isolates a “vital few”variables from a “trivial many” variables.

[0022] An experiment performance component 32 runs the number ofexperiments established for the algorithm by the DoE component 30. Theexperiment performance component 32 comprises a performance metriccomponent 34 that evaluates the results of the experiments run for thealgorithm using a performance metric. The performance metric containsevaluation criteria for comparing the results to a baseline algorithm.Examples of some performance metrics may include the solutionspecificity, sensitivity, robustness, execution time, false positives,false negatives, true positives, true negatives, false classifieds, etc.The baseline algorithm is an algorithm that would provide the best (bysome metric) results if the algorithm undergoing assessment was notavailable. If no baseline algorithm exists, the first path through thenewly developed algorithm (before improvements are being made) can beused as a baseline. The experiment performance component 32 alsocomprises an algorithm adjustment component 36 that adjusts the logic orparameters of the algorithm if the results determined by the performancemetric component 34 are unacceptable. In particular, the algorithmadjustment component 36 changes the algorithm and logic or parameters ifthe results determined by the performance metric component 34 areunacceptable. In this case, the experiment performance component 32 willthen run the experiments for the adjusted algorithm and logic orparameters.

[0023] A module insertion component 38 inserts an additional module intothe algorithm if the results determined by the performance metriccomponent 34 are acceptable. The experiment performance component 32then runs the experiments for the algorithm including the insertedmodule. The module insertion component 38 will continue to add moremodules into the algorithm until performance is satisfactory. Likewise,the experiment performance component 32 will run the experiments for thealgorithm including the inserted modules. This iterative process enablesa designer to assess the performance of the algorithm as each additionalcomponent is inserted.

[0024] A simulation component 40 simulates the behavior of the algorithmusing the results obtained from the experiment performance component 32.The simulation component 40 uses a Monte Carlo simulation to simulatethe behavior of the algorithm. A Monte Carlo simulation is a well-knownsimulation that randomly generates values for uncertain variables overand over to simulate an environment in which a population or studyexists. The Monte Carlo simulation also identifies sources of variation,relationships between covariates and links inputs to outputs. In thisdisclosure, the Monte Carlo simulation component 40 simulates thebehavior of the algorithm on a full design space. In addition, the MonteCarlo simulation component 40 comprises a plurality of confusionmatrices. A confusion matrix contains information indicative ofclassifier performance. In particular, the confusion matrix is definedas an n×n matrix where n is the number of classes considered by theclassifier. The confusion matrix is arranged such that each of the rowscontain true results for a particular class. The columns contain theoutput of the classifier. The resulting matrix shows the correctlyclassified output on the diagonal as the true positives (TP) and truenegatives (TN). If the first class is dedicated to be the normal class(or null fault), then the following definitions can be made: The firstrow except the first entry contains the false positives (FP). The firstcolumn except the first entry contains the false negatives (FN). Theoff-diagonal entries except the first column and the first row containthe falsely classified (FC) classifier results. The confusion matrix canbe normalized by dividing each entry of a row by the sum of the entriesof the respective row. This provides a convenient way to read off theperformance metrics.

[0025] A simulation performance component 44 evaluates the performanceof the simulation for the algorithm. In particular, the simulationperformance component 44 determines if the results of the simulation areacceptable. The criteria for determining whether the simulation resultsare acceptable will vary according to the algorithm and the userperforming the assessment of the algorithm. If the results areacceptable, then the algorithm assessment is over. The simulationperformance component also comprises an algorithm adjustment component46 that adjusts the logic or the parameters of the algorithm if theresults are unacceptable. If the parameters of the algorithm areadjusted, then the simulation component 40 runs the simulation for thesystem including the portion of the adjusted algorithm. The simulationperformance component 44 then evaluates the performance of thissimulation. This process (i.e., adjusting logic or parameters andrunning simulation) continues until the simulation performance component44 determines that the results are acceptable.

[0026]FIG. 3 shows an architectural diagram of a system 48 thatimplements the algorithm performance assessment system 28 shown in FIG.2. In FIG. 3, a computing unit 50 allows a user to access the algorithmperformance assessment system 28. The computing unit 50 can take theform of a hand-held digital computer, personal digital assistantcomputer, notebook computer, personal computer or workstation. The useruses standard output to a window or a web browser 52 such as MicrosoftINTERNET EXPLORER, Netscape NAVIGATOR or Mosaic to locate and displaythe algorithm performance assessment system 28 on the computing unit 50.A communication network 54 such as an electronic or wireless networkconnects the computing unit 50 to the algorithm performance assessmentsystem 28. In particular, the computing unit 50 may connect to thealgorithm performance assessment system 28 through a private networksuch as an extranet or intranet or a global network such as a WAN (e.g.,Internet). As shown in FIG. 3, the algorithm performance assessmentsystem 28 is provided by a server 56, which comprises a web server thatserves the algorithm performance assessment system 28.

[0027] If desired, the system 48 may have functionality that enablesauthentication and access control of users accessing the algorithmperformance assessment system 28. Both authentication and access controlcan be handled at the web server level by the algorithm performanceassessment system 28 itself, or by commercially available packages suchas Netegrity SITEMINDER. Information to enable authentication and accesscontrol such as the user names, location, telephone number,organization, login identification, password, access privileges tocertain resources, physical devices in the network, services availableto physical devices, etc. can be retained in a database directory. Thedatabase directory can take the form of a lightweight directory accessprotocol (LDAP) database; however, other directory type databases withother types of schema may be used including relational databases,object-oriented databases, flat files, or other data management systems.

[0028]FIG. 4 shows a flow chart describing actions performed by thealgorithm performance assessment system 28 shown in FIG. 2 as itoperates within the system 48 shown in FIG. 3. The flow chart begins atblock 58, where an expert determines a design space and selects a subsetof the defined design space at 60 for the algorithm that is undergoingdevelopment. The DoE component 30 then performs the DoE at 62. Asmentioned above, performing the DoE includes establishing an acceptablenumber of experiments and choosing an appropriate level of experimentsfor analyzing the algorithm. Next, the experiment performance component32 runs the number of experiments established for the algorithm at 64.These experiments reflect specific cases of system input behavior whichis then simulated accordingly. The experiment performance component 32then evaluates the results of the experiments run for the algorithm at66. In particular, it evaluates the contribution of the algorithmcomponent under development. This includes using a baseline algorithmprovided at 68 and a performance metric provided at 70 and comparing theresults obtained at 64 to the baseline algorithm and performance metric.

[0029] If the results as determined at 72 are not acceptable, then thealgorithm adjustment component 36 adjusts the algorithm logic or theparameters of the algorithm at 74. The experiment performance component32 then runs the experiments for the adjusted algorithm and logic orparameters at 64. The algorithm performance assessment system 28 thenrepeats blocks 66-72 until the results are acceptable. Next, thealgorithm performance assessment system 28 determines at 76 if there aremore algorithm components that need to be developed. If so, then othercomponents are added to the algorithm at 78. The algorithm performanceassessment system 28 then repeats blocks 64-78 until there are no morecomponents. The decision to develop more algorithm components is made inpart based on the overall performance of the algorithm, availability ofresources and any potentially identified performance improving measures.

[0030] Once it is determined that there are no more components to insertinto the algorithm, then the simulation component 40 simulates thebehavior of the system using the results obtained from the experimentperformance component 32 at 80. As mentioned above, the simulationcomponent 40 uses a Monte Carlo simulation to simulate the behavior ofthe input system. The Monte Carlo simulation uses at least one confusionmatrix obtained from a plurality of matrices at 90. The confusionmatrices are obtained from running experiments at 88 on process data.The Monte Carlo simulation component 40 then uses the confusion matricesto simulate the behavior of the input system. To that end, a z-score iscomputed, which is a statistical measure indicative of how many standarddeviations away from the mean the particular value is located, for agiven fault case. The z-score is computed by interpreting the associatedentries in the confusion matrix as the area under the curve of aGaussian distribution from which the z-score is obtained throughiterative integration. Then, random values are generated based on thez-score which reflect the behavior shown in the confusion matrices. Thesimulation performance component 40 evaluates the performance of thesimulation for the algorithm at 82. In particular, if the simulationperformance component 40 determines that the results of the simulationare acceptable, then the algorithm assessment ends. Otherwise, thealgorithm adjustment component 36 adjusts the logic or parameters of thealgorithm at 84 and modifies the algorithm components at 86. If theparameters or logic or components of the algorithm are adjusted, thenthe simulation is repeated at 80. Blocks 82-86 are repeated until theresults are acceptable.

[0031] The foregoing flow chart of this disclosure shows thefunctionality and operation of the algorithm performance assessmentsystem 28. In this regard, each block represents a module, segment, orportion of code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat in some alternative implementations, the functions noted in theblocks may occur out of the order noted in the figures or, for example,may in fact be executed substantially concurrently or in the reverseorder, depending upon the functionality involved. Furthermore, thefunctions can be implemented in programming languages such as Matlab, C,Visual Basic or JAVA; however, other languages can be used.

[0032] The above-described algorithm performance assessment system 28comprises an ordered listing of executable instructions for implementinglogical functions. The ordered listing can be embodied in anycomputer-readable medium for use by or in connection with acomputer-based system that can retrieve the instructions and hexecutethem. In the context of this application, the computer-readable mediumcan be any means that can contain, store, communicate, propagate,transmit or transport the instructions. The computer readable medium canbe an electronic, a magnetic, an optical, an electromagnetic, or aninfrared system, apparatus, or device. An illustrative, butnon-exhaustive list of computer-readable mediums can include anelectrical connection (electronic) having one or more wires, a portablecomputer diskette (magnetic), a random access memory (RAM) (magnetic), aread-only memory (ROM) (magnetic), an erasable programmable read-onlymemory (EPROM or Flash memory) (magnetic), an optical fiber (optical),and a portable compact disc read-only memory (CDROM) (optical).

[0033] Note that the computer readable medium may comprise paper oranother suitable medium upon which the instructions are printed. Forinstance, the instructions can be electronically captured via opticalscanning of the paper or other medium, then compiled, interpreted orotherwise processed in a suitable manner if necessary, and then storedin a computer memory.

[0034] The algorithm performance assessment system 28 of this disclosureis suitable for assessing the performance of a multitude of algorithmsduring development. One example illustrating an application of thealgorithm performance assessment system 28 is with the development of aninformation fusion tool for aggregating outputs from differentdiagnostic tools. Diagnostic information fusion tools are useful toservice providers and manufactures because of their capability to gatherand combine results obtained from different diagnostic tools. Generally,the information fusion tools maximize the advantages of each diagnostictool, while at the same time minimizing their disadvantages. U.S. patentapplication Ser. No. 09/704,476, filed on Nov. 3, 2000, and entitled“Information Fusion Of Classifiers In Systems With Partial RedundantInformation” provides more information on one type of information fusiontool used to diagnose aircraft engine gas path faults. A description ofhow the algorithm performance assessment system 28 assesses thealgorithms associated with the development of this type of informationfusion tool follows. Note that the algorithm performance assessmentsystem 28 of this disclosure is not limited to algorithms associatedwith information fusion tools and can be used in a variety ofapplications where it is desirable to evaluate the performance of analgorithm during the design process.

[0035]FIG. 5 shows a schematic of a fusion method used to diagnoseaircraft engine gas path faults. In FIG. 5, raw engine measurements areprovided to the system via an input bus 94 which may be a 1553 bus, anARINC bus, or other appropriate data transmission medium. Relevantinformation is sent to the appropriate diagnostic tool such as thosedepicted in diagnostic functions block 96, and to the appropriatesecondary evidence block 98. Outputs from the tools in diagnostic andevidencing blocks 96 and 98 are provided to an information fusion tool100 as lists of fault identifications and performance detection. Theinformation fusion tool 100 aggregates the multiple fault lists into asingle list. This aggregation of fault lists addresses issues such aspartial overlap of output information, missing information, conflictinginformation, different performance levels of the tools inputting theinformation, and the timeliness of the diagnostic data. As can be seen,by FIG. 5, portions of the fusion tool operation will occur while theaircraft powered by the engines under observation are airborne 102,while other actions are taken when the engines are on the ground 104.The operations taking place on the ground provide further refinement ofoutputs by comparing fused fault lists to historic and confirmed listsin order to obtain a final fused output.

[0036] The secondary evidential information block 98 contains existingor known types of information of system operations for gas turbineengines. For example, the model-based engine trending estimator is anexisting piece of software which monitors operation of an engine andmakes predictive statements as to the expected lifetime of the engine'soperation. A vibration analysis detects the amount of vibration withinan engine to determine damage which may be caused by the existence ofexcessive vibration. FADEC fault codes are standardized codes which areknown in the industry to represent states of a system. Intermittentfault code tracking is a mechanism to track the existence of FADEC faultcodes during the operation of a specific system. Thermal historymeasures and records the temperatures reached during operation of anengine.

[0037]FIG. 6 shows a fusion algorithm process map for the informationfusion tool shown in FIG. 5. In FIG. 6 there is an architecture thatincludes eight layers (i.e., layers 0, 1, . . . 7). The first layer is aconcurrency layer 106, which is a pre-processing step to the main fusionoperations. The concurrency layer 106 uses a temporal aggregation module108 to perform aggregation of information over time and collectinformation before the main information fusion process is triggered. Inaddition, the concurrency layer 106 uses a decision fading module 110 toperform an information fading operation when certain temporal conditionsare met. An association layer 112 exploits information about preferredmisclassifications of a classification tool using a correlating module114. A scaling layer 116 uses a reliability forming module 118 andrelevance scaling module 120 to integrate a priori information regardingthe performance of a classification tool. A strengthening layer 122reinforces an opinion about a class when several classification toolsindicate the same state of a class, such as the existence of a fault.The strengthening layer 122 uses a reinforcement module 124 toaccomplish this process. A weakening layer 126 discounts information ifseveral tools disagree on the classification state and it also discountsinformation if any classification tool indicates several classes at thesame time, by use of a few tools discount module 128 and a many faultsdiscount module 130.

[0038] An evidence updating layer 132 incorporates secondary evidentialinformation, which is not generated by a classification tool, via anevidential reinforcement module 134. This type of information is usedonly to support a classification state, not to discount a classificationstate. A tie-breaking layer 136 deals with high ranking system states(e.g., faults) having similar confidence levels. It employs twostrategies: one is to provide a higher weight value to a class thattraditionally occurs more often (i.e., a frequency module 140); theother is to weigh a class state (e.g., a fault) higher if it is morecritical to successful operation of the system (i.e., a criticallymodule 138). A backscaling layer 142 performs a transformation by usinga backscaling module 144, which makes the fused informationinterpretable to an end user. U.S. patent application Ser. No.09/704,476, filed on Nov. 3, 2000, and entitled “Information Fusion OfClassifiers In Systems With Partial Redundant Information” provides moreinformation on this fusion algorithm process map.

[0039] A description of how the algorithm performance assessment system28 can be used to design an information fusion tool such as the oneshown in FIGS. 5 and 6 follows. To design this type of informationfusion tool one should establish that each heuristic implemented in thelayered architecture contributes to an improvement of the result. Tothat end, a designer can break down the development process into smallersteps and adapt a strategy of complete design cycles for each heuristic.That is, each step encompasses iterations of conception, implementation,and testing. This development process reduces the re-design necessaryafter completion of the whole fusion architecture, which at that stagewould be more difficult because the influence of individual heuristicswould be masked in the overall architecture. At the onset, it is notclear what effect the addition of new modules of the algorithm will haveon the overall performance and whether the chosen implementation of thatheuristic will move the error into the desired direction. Similarly, theparameters are not known beforehand and at least a coarse tuning isneeded to be carried out for each heuristic.

[0040] The first step in using the algorithm performance assessmentsystem 28 to evaluate the design of such an information fusion tool isto have the designer determine the problem space and select a subsetfrom this space. In this example, assume that the designer selects 2diagnostic tools, 1 evidential tool and 2 possible fault states from aproblem space of 3 diagnostic tools, 5 evidential tools and 7 faultstates with continuous-valued variables. Each one of the states in thesubset is then limited to take on 3 possible confidence values. Thisamounts to 729 solutions. Even for a very constrained problem such asthis one, 729 is a number too large for expert evaluation and hence willbe reduced using the DoE. In this example, a half factorial DoE ispreferable for providing a conceptual set of faults for the entiredesign space. In particular, the half factorial DoE cuts 729 solutionsto 45 experiments. Table 1 shows the results of the 45 experiments andthe results of expert evaluation. TABLE 1 DoE set and expert specifiedtarget output Tool A Tool B Evidence Tool A Tool B Evidence Fault 1Fault 1 Fault 1 Fault 2 Fault 2 Fault 2 Target 1 Target 2 0 0 0 0 0 0 00 1 0 0 0 0 1 0.6 0 0 1 0 0 0 1 0.3 0 1 1 0 0 0 0 0.9 0 0 0 1 0 0 1 0 01 0 1 0 0 0 0.8 0 0 1 1 0 0 0 0.4 0 1 1 1 0 0 1 1 0 0 0 0 1 0 1 0 0.1 10 0 1 0 0 0.5 0 0 1 0 1 0 0 0.3 0 1 1 0 1 0 1 0.9 0.1 0 0 1 1 0 0 0 0 10 1 1 0 1 0.5 0 0 1 1 1 0 1 0.3 0 1 1 1 1 0 0 1 0 0 0 0 0 1 1 0 0.3 1 00 0 1 0 0.5 0.2 0 1 0 0 1 0 0.2 0.1 1 1 0 0 1 1 0.8 0.2 0 0 1 0 1 0 00.2 1 0 1 0 1 1 0.5 0.1 0 1 1 0 1 1 0.3 0.2 1 1 1 0 1 0 0.9 0.1 0 0 0 11 0 0 0.6 1 0 0 1 1 1 0.4 0.5 0 1 0 1 1 0 0 0.4 1 1 0 1 1 0 0.6 0.4 0 01 1 1 1 0 0.7 1 0 1 1 1 0 0.5 0.5 0 1 1 1 1 0 0.2 0.5 1 1 1 1 1 1 0.80.5 0 0.5 0.5 0.5 0.5 0.5 0.3 0.3 1 0.5 0.5 0.5 0.5 0.5 0.7 0.3 0.5 00.5 0.5 0.5 0.5 0.3 0.3 0.5 1 0.5 0.5 0.5 0.5 0.5 0.2 0.5 0.5 0 0.5 0.50.5 0.4 0.3 0.5 0.5 1 0.5 0.5 0.5 0.4 0.3 0.5 0.5 0.5 0 0.5 0.5 0.4 0.20.5 0.5 0.5 1 0.5 0.5 0.4 0.4 0.5 0.5 0.5 0.5 0 0.5 0.4 0.3 0.5 0.5 0.50.5 1 0.5 0 0.4 0.5 0.5 0.5 0.5 0.5 0 0.4 0.3 0.5 0.5 0.5 0.5 0.5 1 0.40.3 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.3

[0041] The columns labeled “Target 1” and “Target 2” contain the resultof the expert evaluation, which represents the diagnosis for Fault 1 andFault 2, respectively. The DoE in this example is defined as a“bottom-up DoE” because the diagnostic expert is exposed only to thetool output without knowledge of the faults.

[0042] Next, the algorithm performance assessment system 28 evaluatesthe results of the experiments using a performance metric and a baselinealgorithm. In this example, an index aggregating the weighted componentsof false positives, false negatives, and false classifieds, or any othersuitable metric relative to a baseline algorithm can be used as theperformance metric, however, other metrics such as sum squared error(SSE), total error rate (TER), total success rate (TSR), can be used.The baseline algorithm may comprise a benchmark algorithm that performeda maximum wins strategy on transformed and normalized input scaled bythe a priori reliability of the diagnostic tool. In addition, an overallperformance index that weighs the individual components false positives(FP), false negatives (FN), and false classified (FC) is used. Theweightings of the individual components are driven by the followingequation:

0.6*(1−FP)+0.3*(1−FN)+0.1*(1−FC)

[0043] In this example, the benchmark performance index is set to zero.Therefore, an increase in performance is measured as the fraction ofimprovement from that baseline to perfect performance, expressed inpercent. The development process comprises the iterative addition of newalgorithm components and refinement of existing components. In thisexample, the layers in the information fusion architecture roughlyrepresent the final set of components. Each addition or change is thensubjected to the evaluation process. Only components that improve theperformance index are included in the overall architecture, thusallowing immediate evaluation. This iterative evaluation processcontinues until all the results for all of the components of thealgorithm are acceptable.

[0044] Once all the results are acceptable the algorithm performanceassessment system 28 simulates the actual system fault states (“topdown”) for the full input set with 3 diagnostic tools, 5 evidentialtools, and 7 fault states to fine tune the algorithm and parameters. TheMonte Carlo simulation allows each fault state to take on any valuebetween 0 and 1 per architecture design requirements. The confusionmatrices are used to create a system response that is indicative ofrealistic tool behavior. Table 2 shows an example of a confusion matrixthat can be used in the simulation of the information fusion tool. TABLE2 Confusion Matrix {circumflex over (F)}₀ {circumflex over (F)}₁{circumflex over (F)}₂ {circumflex over (F)}₃ {circumflex over (F)}₄{circumflex over (F)}₅ {circumflex over (F)}₆ F₀ 0.832 0.023 0.039 0.0350.035 0.013 0.023 F₁ 0.258 0.696 0.019 0.012 0.005 0.005 0.005 F₂ 0.3130.011 0.582 0.029 0.027 0.014 0.024 F₃ 0.325 0.010 0.029 0.573 0.0520.007 0.004 F₄ 0.382 0.007 0.027 0.041 0.495 0.007 0.041 F₅ 0.094 0.0010.013 0.005 0.012 0.847 0.028 F₆ 0.234 0.007 0.032 0.004 0.058 0.0260.639

[0045] In this example, the confusion matrices of the diagnostic toolsare the primary sources of a priori information for the informationfusion tool. As a performance measure for the individual diagnostictools, the confusion matrices list the observed faults versus theestimated faults. Because all faults are enumerated, it is possible toobtain information not only about the correctly classified states, butalso about the false positives (FP), false negatives (FN), and falseclassified (FC) states. In Table 2, the rows list the actual faults, thecolumns list the estimated faults. Note that the fault F₀ represents theno fault condition. The diagonal entries of the confusion matrixrepresent the correctly classified cases. The first row, except thefirst entry, contains the FP state. The first column, except the firstentry, contains the FN state. The off-diagonal elements, except the FPand FN states, are the FC states. Table 2 also shows the normalizedconfusion matrix for a diagnostic tool where each entry of a row wasdivided by the number of experiments for each class (i.e., the sum ofthe entries of the respective rows). The faults are denoted as F_(n)where n={0, . . . , 6}.

[0046] The Monte Carlo simulation in this example generally comprisesthree parts. One part includes computing the z-score, which as mentionedabove, is a statistical measure indicative of how many standarddeviations away from the mean the particular value is located, for agiven a fault case. The z-score is computed by interpreting theassociated entries in the confusion matrix as the area under the curveof a Gaussian distribution from which the z-score is obtained throughiterative integration. Another part of the Monte Carlo simulationincludes assigning random values based on the z-score. As an example,assume that the reliability of a diagnostic tool for a particular faultis 95%. This value is obtained from the diagonal entries of theconfusion matrix, e.g., 0.95. The interpretation is that the diagnostictool classifies correctly in 95% of the cases. Therefore, 95% of theoutput cases are generated between 0.5 and 1 and 5% of the output casesare generated between 0 and 0.5. FIG. 7 illustrates an example curvethat can be used to simulate a diagnostic tool output. The third part ofthe simulation is to carry out the system run, in this case theinformation fusion. These parts of the Monte Carlo simulation arerepeated many times until statistically meaningful results have beenobtained. The results are then compiled in confusion matrix format.

[0047] After performing the simulation, the simulation performancecomponent 44 determines if the results are acceptable. The results areacceptable if the performance index is greater than a predeterminedthreshold. If necessary, the logic or parameters and/or algorithm for aparticular heuristic in the algorithm are adjusted. The process ofdetermining if the simulation results are acceptable is repeated until adesired system response is obtained.

[0048] It is apparent that there has been provided in accordance withthis invention, an algorithm performance assessment system, method andcomputer product. While the invention has been particularly shown anddescribed in conjunction with a preferred embodiment thereof, it will beappreciated that variations and modifications can be effected by aperson of ordinary skill in the art without departing from the scope ofthe invention.

1. A system for assessing the performance of an algorithm duringdevelopment, comprising: a design of experiments component thatestablishes an acceptable number of experiments for analyzing thealgorithm; an experiment performance component that runs the establishednumber of experiments for the algorithm; and a simulation component thatsimulates the behavior of the algorithm using results from theexperiment performance component.
 2. The system according to claim 1,wherein the design of experiments component selects an appropriate levelof experiments for analyzing the algorithm.
 3. The system according toclaim 2, wherein the design of experiments component selects theappropriate level of experiments from at least one of full factorialexperiments, fractional factorial experiments or screening experiments.4. The system according to claim 1, wherein the experiment performancecomponent comprises a performance metric component that evaluates theresults of the experiments run for the algorithm.
 5. The systemaccording to claim 4, wherein the experiment performance componentcomprises an algorithm adjustment component that adjusts logic orparameters of the algorithm for unacceptable results.
 6. The systemaccording to claim 5, wherein the experiment performance component runsthe established number of experiments for the algorithm after adjustmentof logic or parameters.
 7. The system according to claim 1, furthercomprising a module insertion component that inserts a module into thealgorithm.
 8. The system according to claim 7, wherein the experimentperformance component runs the established number of experiments for thealgorithm including the inserted module.
 9. The system according toclaim 1, wherein the simulation component performs a Monte Carlosimulation.
 10. The system according to claim 1, wherein the simulationcomponent uses at least one confusion matrix to simulate the behavior ofthe algorithm.
 11. The system according to claim 1, further comprising asimulation performance component that evaluates the performance of thesimulation of the algorithm.
 12. The system according to claim 11,wherein the simulation performance component comprises an algorithmadjustment component that adjusts logic or parameters of the algorithmfor unacceptable results.
 13. The system according to claim 12, whereinthe simulation component runs the simulation for the algorithm afteradjustment of logic or parameters.
 14. A system for assessing theperformance of an algorithm during development, comprising: a design ofexperiments component that establishes an acceptable number ofexperiments for analyzing the algorithm; an experiment performancecomponent that runs the established number of experiments for algorithm;a simulation component that simulates the behavior of the algorithmusing results from the experiment performance component; and asimulation performance component that evaluates the performance of thesimulation for the algorithm.
 15. The system according to claim 14,wherein the design of experiments component selects an appropriate levelof experiments for analyzing the algorithm.
 16. The system according toclaim 15, wherein the design of experiments component selects theappropriate level of experiments from at least one of full factorialexperiments, fractional factorial experiments or screening experiments.17. The system according to claim 14, wherein the experiment performancecomponent comprises a performance metric component that evaluates theresults of the experiments run for the algorithm.
 18. The systemaccording to claim 17, wherein the experiment performance componentcomprises an algorithm adjustment component that adjusts logic orparameters of the algorithm for unacceptable results.
 19. The systemaccording to claim 18, wherein the experiment performance component runsthe established number of experiments for the portion of the algorithmafter adjustment of parameters.
 20. The system according to claim 14,further comprising a module insertion component that inserts a moduleinto the algorithm.
 21. The system according to claim 20, wherein theexperiment performance component runs the established number ofexperiments for the algorithm including the inserted module.
 22. Thesystem according to claim 14, wherein the simulation component performsa Monte Carlo simulation.
 23. The system according to claim 14, whereinthe simulation component uses at least one confusion matrix to simulatethe behavior of the algorithm.
 24. The system according to claim 14,wherein the simulation performance component comprises an algorithmadjustment component that adjusts logic or parameters of the algorithmfor unacceptable results.
 25. The system according to claim 24, whereinthe simulation component runs the simulation for the algorithm afteradjustment of logic or parameters.
 26. A system for assessing theperformance of an algorithm during development, comprising: a design ofexperiments component that establishes an acceptable number ofexperiments for analyzing the algorithm; an experiment performancecomponent that runs the established number of experiments for thealgorithm and uses a performance metric to evaluate the results of theexperiments; a Monte Carlo simulation component that simulates thebehavior of the algorithm using results from the experiment performancecomponent with a Monte Carlo simulation; and a simulation performancecomponent that evaluates the performance of the Monte Carlo simulationfor the algorithm.
 27. The system according to claim 26, wherein thedesign of experiments component selects an appropriate level ofexperiments from at least one of full factorial experiments, fractionalfactorial experiments or screening experiments.
 28. The system accordingto claim 26, wherein the experiment performance component comprises analgorithm adjustment component that adjusts logic or parameters of thealgorithm for unacceptable results.
 29. The system according to claim28, wherein the experiment performance component runs the establishednumber of experiments for the algorithm after adjustment of logic orparameters.
 30. The system according to claim 26, wherein the MonteCarlo simulation component uses at least one confusion matrix tosimulate the behavior of the algorithm.
 31. The system according toclaim 26, wherein the simulation performance component comprises analgorithm adjustment component that adjusts logic or parameters of thealgorithm for unacceptable results.
 32. The system according to claim31, wherein the simulation component runs the simulation for thealgorithm after adjustment of parameters.
 33. A system for assessing theperformance of an algorithm during development, comprising: a design ofexperiments component that establishes an acceptable number ofexperiments for analyzing the algorithm; an experiment performancecomponent that runs the established number of experiments for thealgorithm; a performance metric component that evaluates the results ofthe experiments run for the algorithm; and an algorithm adjustmentcomponent that adjusts logic or parameters of the algorithm forunacceptable results.
 34. The system according to claim 33, wherein thedesign of experiments component selects an appropriate level ofexperiments for analyzing the algorithm from at least one of fullfactorial experiments, fractional factorial experiments or screeningexperiments.
 35. The system according to claim 33, wherein theexperiment performance component runs the established number ofexperiments for the algorithm after adjustment of logic or parameters.36. The system according to claim 33, further comprising a moduleinsertion component that inserts a module into the algorithm.
 37. Thesystem according to claim 36, wherein the experiment performancecomponent runs the established number of experiments for the algorithmincluding the inserted module.
 38. A system for assessing theperformance of an algorithm during development, comprising: a MonteCarlo simulation component that simulates the behavior of the algorithmwith a Monte Carlo simulation, wherein the Monte Carlo simulationcomponent uses at least one confusion matrix to simulate the behavior ofthe algorithm; a simulation performance component that evaluates theperformance of the Monte Carlo simulation for the algorithm; and analgorithm adjustment component that adjusts logic or parameters of thealgorithm for unacceptable results.
 39. The system according to claim38, wherein the Monte Carlo simulation component runs the simulation forthe algorithm after adjustment of logic or parameters.
 40. A method forassessing the performance of an algorithm during development,comprising: using design of experiments to establish an acceptablenumber of experiments for analyzing the algorithm; running theestablished number of experiments for the algorithm; and simulating thebehavior of the algorithm using results from the experiments.
 41. Themethod according to claim 40, wherein the using of design of experimentscomprises selecting an appropriate level of experiments for analyzingthe algorithm.
 42. The method according to claim 41, wherein theappropriate level of experiments comprise at least one of full factorialexperiments, fractional factorial experiments or screening experiments.43. The method according to claim 40, further comprising using aperformance metric to evaluate the results of the experiments run forthe algorithm.
 44. The method according to claim 43, further comprisingadjusting logic or parameters of the algorithm for unacceptable results.45. The method according to claim 44, further comprising running theestablished number of experiments for the algorithm after adjustment oflogic or parameters.
 46. The method according to claim 40, furthercomprising inserting a module into the algorithm.
 47. The methodaccording to claim 46, further comprising running the established numberof experiments for the algorithm including the inserted module.
 48. Themethod according to claim 40, wherein the simulating comprises using atleast one confusion matrix to simulate the behavior of the algorithm.49. The method according to claim 40, further comprising evaluating theperformance of the simulation for the algorithm.
 50. The methodaccording to claim 49, further comprising adjusting logic or parametersof the algorithm for unacceptable results.
 51. The method according toclaim 50, further comprising running the simulation for the algorithmafter adjustment of logic or parameters.
 52. A method for assessing theperformance of an algorithm during development, comprising: using designof experiments to establish an acceptable number of experiments foranalyzing the algorithm; running the established number of experimentsfor the algorithm; simulating the behavior of the algorithm usingresults from the experiments; and evaluating the performance of thesimulation for the algorithm.
 53. The method according to claim 52,wherein the using of design of experiments comprises selecting anappropriate level of experiments for analyzing the algorithm from atleast one of full factorial experiments, fractional factorialexperiments or screening experiments.
 54. The method according to claim52, further comprising using a performance metric to evaluate theresults of the experiments run for the algorithm.
 55. The methodaccording to claim 54, further comprising adjusting logic or parametersof the algorithm for unacceptable results.
 56. The method according toclaim 55, further comprising running the established number ofexperiments for the algorithm after adjustment of logic or parameters.57. The method according to claim 52, further comprising inserting amodule into the algorithm.
 58. The method according to claim 57, furthercomprising running the established number of experiments for thealgorithm including the inserted module.
 59. The method according toclaim 52, wherein the simulation is a Monte Carlo simulation.
 60. Themethod according to claim 52, wherein the simulating uses at least oneconfusion matrix to simulate the behavior of the algorithm.
 61. Themethod according to claim 52, further comprising adjusting logic orparameters of the algorithm for unacceptable results.
 62. The methodaccording to claim 61, further comprising running the simulation for thealgorithm after adjustment of logic or parameters.
 63. A method forassessing the performance of an algorithm during development,comprising: using a design of experiments to establish an acceptablenumber of experiments for analyzing the algorithm; running theestablished number of experiments for the algorithm; using a performancemetric to evaluate the results of the experiments; using a Monte Carlosimulation to simulate the behavior of the algorithm using results fromthe performance metric; and evaluating the performance of the MonteCarlo simulation for the algorithm.
 64. The method according to claim63, wherein using the design of experiments comprises selecting anappropriate level of experiments from at least one of full factorialexperiments, fractional factorial experiments or screening experiments.65. The method according to claim 63, further comprising adjusting logicor parameters of the algorithm for unacceptable results.
 66. The methodaccording to claim 65, further comprising running the established numberof experiments for the algorithm after adjustment of logic orparameters.
 67. The method according to claim 63, wherein the simulatingcomprises using at least one confusion matrix to simulate the behaviorof the algorithm.
 68. The method according to claim 63, furthercomprising adjusting logic or parameters of the algorithm forunacceptable results.
 69. The method according to claim 68, furthercomprising running the simulation for the algorithm after adjustment oflogic or parameters.
 70. A method for assessing the performance of analgorithm during development, comprising: using a design of experimentsto establish an acceptable number of experiments for analyzing thealgorithm; running the established number of experiments for thealgorithm; evaluating the results of the experiments run for thealgorithm with a performance metric; and adjusting logic or parametersof the algorithm for unacceptable results.
 71. The method according toclaim 70, wherein the using of design of experiments comprises selectingan appropriate level of experiments for analyzing the algorithm from atleast one of full factorial experiments, fractional factorialexperiments or screening experiments.
 72. The method according to claim70, further comprising running the established number of experiments forthe algorithm after adjustment of parameters.
 73. The method accordingto claim 70, further comprising inserting a module into the algorithm.74. The method according to claim 73, further comprising running theestablished number of experiments for the algorithm including theinserted module.
 75. A method for assessing the performance of analgorithm during development, comprising: simulating the behavior of thealgorithm with a Monte Carlo simulation, wherein the Monte Carlosimulation uses at least one confusion matrix to simulate the behaviorof the algorithm; evaluating the performance of the Monte Carlosimulation for the algorithm; and adjusting logic or parameters of thealgorithm for unacceptable results.
 76. The method according to claim75, further comprising running the Monte Carlo simulation for thealgorithm after adjustment of logic or parameters.
 77. Acomputer-readable medium storing computer instructions for instructing acomputer system to assess the performance of an algorithm duringdevelopment, the computer instructions comprising: using design ofexperiments to establish an acceptable number of experiments foranalyzing the algorithm; running the established number of experimentsfor the algorithm; and simulating the behavior of the algorithm usingresults from the experiments.
 78. The computer-readable medium accordingto claim 77, wherein the using of design of experiments comprisesinstructions for selecting an appropriate level of experiments foranalyzing the algorithm.
 79. The computer-readable medium according toclaim 78, wherein the appropriate level of experiments comprise at leastone of full factorial experiments, fractional factorial experiments orscreening experiments.
 80. The computer-readable medium according toclaim 77, further comprising instructions for using a performance metricto evaluate the results of the experiments run for the algorithm. 81.The computer-readable medium according to claim 80, further comprisinginstructions for adjusting logic or parameters of the algorithm forunacceptable results.
 82. The computer-readable medium according toclaim 81, further comprising instructions for running the establishednumber of experiments for the algorithm after adjustment of logic orparameters.
 83. The computer-readable medium according to claim 77,further comprising instructions for inserting a module into thealgorithm.
 84. The computer-readable medium according to claim 83,further comprising instructions for running the established number ofexperiments for the algorithm including the inserted module.
 85. Thecomputer-readable medium according to claim 77, wherein the simulatingcomprises instructions for using at least one confusion matrix tosimulate the behavior of the algorithm.
 86. The computer-readable mediumaccording to claim 77, further comprising instructions for evaluatingthe performance of the simulation for the algorithm.
 87. Thecomputer-readable medium according to claim 86, further comprisinginstructions for adjusting logic or parameters of the algorithm forunacceptable results.
 88. The computer-readable medium according toclaim 87, further comprising instructions for running the simulation forthe algorithm after adjustment of logic or parameters.
 89. Acomputer-readable medium storing computer instructions for instructing acomputer system to assess the performance of an algorithm duringdevelopment, the computer instructions comprising: using design ofexperiments to establish an acceptable number of experiments foranalyzing the algorithm; running the established number of experimentsfor the algorithm; simulating the behavior of the algorithm usingresults from the experiments; and evaluating the performance of thesimulation for the algorithm.
 90. The computer-readable medium accordingto claim 89, wherein the using of design of experiments comprisesinstructions for selecting an appropriate level of experiments foranalyzing the algorithm from at least one of full factorial experiments,fractional factorial experiments or screening experiments.
 91. Thecomputer-readable medium according to claim 89, further comprisinginstructions for using a performance metric to evaluate the results ofthe experiments run for the algorithm.
 92. The computer-readable mediumaccording to claim 91, further comprising instructions for adjustinglogic or parameters of the algorithm for unacceptable results.
 93. Thecomputer-readable medium according to claim 92, further comprisinginstructions for running the established number of experiments for thealgorithm after adjustment of logic or parameters.
 94. Thecomputer-readable medium according to claim 89, further comprisinginstructions for inserting a module into the algorithm.
 95. Thecomputer-readable medium according to claim 94, further comprisinginstructions for running the established number of experiments for thealgorithm including the inserted module.
 96. The computer-readablemedium according to claim 89, wherein the simulation is a Monte Carlosimulation.
 97. The computer-readable medium according to claim 89,wherein the simulating uses at least one confusion matrix to simulatethe behavior of the algorithm.
 98. The computer-readable mediumaccording to claim 89, further comprising instructions for adjustinglogic or parameters of the algorithm for unacceptable results.
 99. Thecomputer-readable medium according to claim 98, further comprisinginstructions for running the simulation for the algorithm afteradjustment of logic or parameters.
 100. A computer-readable mediumstoring computer instructions for instructing a computer system toassess the performance of an algorithm during development, the computerinstructions comprising: using a design of experiments to establish anacceptable number of experiments for analyzing the algorithm; runningthe established number of experiments for the algorithm; using aperformance metric to evaluate the results of the experiments; using aMonte Carlo simulation to simulate the behavior of the algorithm usingresults from the performance metric; and evaluating the performance ofthe Monte Carlo simulation for the he algorithm.
 101. Thecomputer-readable medium according to claim 100, wherein using thedesign of experiments comprises instructions for selecting anappropriate level of experiments from at least one of full factorialexperiments, fractional factorial experiments or screening experiments.102. The computer-readable medium according to claim 100, furthercomprising instructions for adjusting logic or parameters of thealgorithm for unacceptable results.
 103. The computer-readable mediumaccording to claim 102, further comprising instructions for running theestablished number of experiments for the algorithm after adjustment oflogic or parameters.
 104. The computer-readable medium according toclaim 100, wherein the simulating comprises instructions for using atleast one confusion matrix to simulate the behavior of the algorithm.105. The computer-readable medium according to claim 100, furthercomprising instructions for adjusting logic or parameters of thealgorithm for unacceptable results.
 106. The computer-readable mediumaccording to claim 105, further comprising instructions for running thesimulation for the algorithm after adjustment of logic or parameters.107. A computer-readable medium storing computer instructions forinstructing a computer system to assess the performance of an algorithmduring development, the computer instructions comprising: using a designof experiments to establish an acceptable number of experiments foranalyzing the algorithm; running the established number of experimentsfor the algorithm; evaluating the results of the experiments run for thealgorithm with a performance metric; and adjusting logic or parametersof the algorithm for unacceptable results.
 108. The computer-readablemedium according to claim 107, wherein the using of design ofexperiments comprises instructions for selecting an appropriate level ofexperiments for analyzing the algorithm from at least one of fullfactorial experiments, fractional factorial experiments or screeningexperiments.
 109. The computer-readable medium according to claim 107,further comprising instructions for running the established number ofexperiments for the algorithm after adjustment of logic or parameters.110. The computer-readable medium according to claim 107, furthercomprising instructions for inserting a module into the algorithm. 111.The computer-readable medium according to claim 110, further comprisinginstructions for running the established number of experiments for thealgorithm including the inserted module.
 112. A computer-readable mediumstoring computer instructions for instructing a computer system toassess the performance of an algorithm during development, the computerinstructions comprising: simulating the behavior of the algorithm with aMonte Carlo simulation, wherein the Monte Carlo simulation uses at leastone confusion matrix to simulate the behavior of the algorithm;evaluating the performance of the Monte Carlo simulation for thealgorithm; and adjusting logic or parameters of the algorithm forunacceptable results.
 113. The computer-readable medium according toclaim 12, further comprising instructions for running the Monte Carlosimulation for the algorithm after adjustment of logic or parameters.